Generic Detection of Code Injection Attacks using Network-level Emulation

Code injection attacks against server and client applications have become the primary method of malware spreading. A promising approach for the detection of previously unknown code injection attacks at the network level, irrespective of the particular exploitation method used or the vulnerability being exploited, is to identify the malicious code that is part of the attack vector, also known as shellcode. Initial implementations of this approach attempt to identify the presence of shellcode in network inputs using detection algorithms based on static code analysis. However, static analysis cannot effectively handle malicious code that employs advanced obfuscation methods such as anti-disassembly tricks or self-modifying code, and thus these detection methods can be easily evaded. In this dissertation we present network-level emulation, a generic code injection attack detection method based on dynamic code analysis using emulation. Our prototype attack detection system, called Nemu, uses a CPU emulator to dynamically analyze valid instruction sequences in the inspected traffic. Based on runtime behavioral heuristics, the system identifies inherent patterns exhibited during the execution of the shellcode, and thus can detect the presence of malicious code in arbitrary inputs. We have developed heuristics that cover the most widely used shellcode types, including self-decrypting and non-self-contained polymorphic shellcode, plain or metamorphic shellcode, and memory-scanning shellcode. Network-level emulation does not rely on any exploit or vulnerability specific signatures, which allows the detection of previously unknown attacks. At the same time, the actual execution of the attack code on a CPU emulator makes the detector robust to evasion techniques like indirect jumps and self-modifications. Furthermore, each input is inspected autonomously, which makes the approach effective against targeted attacks. Our experimental evaluation with publicly available shellcode construction engines, attack toolkits, and real attacks captured in the wild, shows that Nemu is more robust to obfuscation techniques compared to previous proposals, while it can effectively detect a broad range of different shellcode implementations without any prior exploit-specific information. At the same time, extensive testing using benign generated and real data did not produce any false positives. To assess the effectiveness of our approach under realistic conditions we deployed Nemu in several production networks. Over the course of more than one year of continuous operation, Nemu detected more than 1.2 million attacks against real systems. We provide a thorough analysis of the captured attacks, focusing on the structure and operation of the shellcode, as well as the overall attack activity in relation to the different targeted services. The large and diverse set of the detected attacks combined with the zero false positive rate over the whole monitoring period demonstrate the effectiveness and practicality of our approach. Finally, we identify challenges faced by existing network trace anonymization schemes for safely sharing attack traces that contain self-decrypting shellcode. To alleviate this problem, we present an anonymization method that identifies and properly sanitizes sensitive information contained in the encrypted part of the shellcode that is otherwise not exposed on the wire. Thesis Advisor: Prof. Evangelos Markatos